I'm using node-fetch to try to get the contents of a website. I have read a couple similar questions like this one or this one but I still can't figure it out.
When I'm on the page, I see one set of HTML when I go to View:Source and another in the Inspector. Seems like this is because the website is showing me the instantaneous DOM whereas View Source (CTRL+U) shows me what was initially sent? For example, the "View:Source" of the HTML begins:
<!doctype html><html lang="en" translate="no"><head><meta name="version"/><meta charset="utf-8"/><meta name="viewport" content="width=device-width,initial-scale=1,shrink-to-fit=no"/>
And the "Inspector" version of the HTML begins:
<html translate="no" class="fontawesome" lang="en"><head style=""><script
Here is how my request is currently set up using node-fetch:
const fetch = (...args) => import('node-fetch').then(({default: fetch}) => fetch(...args));
var options1 = {
method: 'POST'
,headers:{/*omitting the options here*/ }
,body: new URLSearchParams(postData)};
var urlString1 = new URL(url);
fetch(urlString1, options1)
.then(res =>{console.log(res.headers); return res.text();})
.then(values=>{ console.log(values);});
;
How do I set up my request to get the HTML from the "Inspector" not from "View Source"?
Seems like this is because the website is showing me the instantaneous DOM whereas View Source (CTRL+U) shows me what was initially sent?
Exactly.
View Source is showing you the text content of the HTML.
The DOM tree in your browser's developer tools (what you see when you right-click and click Inspect Element) is showing you what the elements actually loaded are, including any manipulation that's been done with JavaScript since loading. Basically, the HTML is parsed, the tree is built, and then for developer convenience, it's converted back to HTML again in a nice formatted way for you to look at in Developer Tools.
How do I set up my request to get the HTML from the "Inspector" not from "View Source"?
You need to actually run a browser engine, such as headless Chrome. There are several NPM packages for doing this.